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Learning/Performance 

Challenges 

Human  Challenges 

•  Demands  on  individuals 

•  From  learning  perspective:  complex  skills  and  knowledge, 
limited  time,  lack  of  practice  to  develop  automaticity 

Solution  Challenges 

•  Recurrent  theme — getting  more  out  of  the  people 

•  Need  to  do  more  with  resources — effective,  efficient 

•  Need  to  get  engaged  with  other  and  learning  activities/job 
tasks 

•  Target  all  three  characteristics — Effective,  Effective, 
Engaging 

Designer  Challenges 

•  Differences  between  Learning  and  Performance  Strategies 


Workplace  Challenges 

Technology  Progress/Workplace  Needs 

1 .  Scientific/technology  progress  places  new  demands  on  workforce 

2.  Rapid  technology  advances  require  workforce  to  have  ability  to  adapt  to 
changing  technologies 

3.  Technological  progress  has  an  impact  not  only  on  how  we  do  things,  but 
also  how  we  think 

a.  “Knowing”  was  once — being  able  to  remember  and  repeat  information 

b.  “Knowing”  is  now — being  able  to  find  and  use  information 

4.  Current  work  tasks  favor  strong  non-routine  cognitive  skills  such  as 
problem  solving,  abstract  reasoning,  communication,  and  collaboration 

5.  Learning  goals  need  to  focus  towards  helping  students  develop  learning 
strategies  and  cognitive  tools 

6.  Students  need  basic  understanding  about  key  subjects  but  also  need  to 
learn  how  to  ask  meaningful  questions 

7.  Many  work  tasks  are  too  complex  and  too  large  to  be  handled  by  a 
single  individual 


Viable  Solution 


1  .Teams 

a.  Formed  for  collective  resources  (e.g.,  knowledge,  skills, 
and  diverse  expertise)  are  required  to  complete  and  solve 
complex  tasks 

2.Team  Research — Understanding  Team  Behaviors 

a. Teams  work  effectively  together  when  they  think  in  similar 
ways 

b. Key  indicator  of  team  performance — degree  team 
members  share  similar  conceptualizations  of  problems  and 
approaches 

c.  Research  had  led  to  improved  understanding  about  team 
behaviors 

d.  Theoretical  constructs  (e.g.,  shared  understanding, 
distributed  cognition,  shared  knowledge,  shared  cognition) 
have  been  developed  to  better  understand  team 


Broad  Impact — Benefits  of 
Theoretical  Framework 

1.  Developing  and  Improving  Teams 

a. Use  a  team  cognition  framework  to  systematically  study  aspects 
of  instructional  and  performance  strategies  that  affect  team 
performance  in  complex  problem-solving  domains 

b. Theoretical  framework  can  guide  the  design  of  effective  learning 
activities  (interaction  strategies) 

c. Theoretical  framework  can  inform  our  knowledge  of  teaming 
processes 

d. Working  in  teams  has  more  benefits  than  just  job  performance 

2.  Using  Teams  for  Learning 

a. Few  instructional  methods  use  teams  as  a  salient  characteristic 

b. Team  Based  Learning  (TBL)  specific  instructional  method 


Degree  of  Information  Connectivity 
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Adapted  from — Ifenthaler,  D., 
Pirnay-Dummer,  P.,  &  Seel,  N.  M. 
(2010)  Computer-based 
diagnostics  and  systematic 
analysis  of  knowledge.  New  York: 
Springer 
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Online  Team  Challenges 

1.  Forming 

2.  Roles 

3.  Expectations 

4.  Communication 

5.  Style 

6.  Intent 

7.  Feedback 
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Online  Team  Solutions 

Solutions  approach  informed 

•  Anecdotal  Data 

•  Evaluation  Data  Driven  Decision-Making 

•  Modeling 

•  Experimental  Research 
Techniques  for  becoming  informed 

•  Advanced/Emergent  Performance  (Learning) 


Assessment 
TAD  I— HI  MATT— ABM 
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Online  Team  Practice 

1 .  Development  Needs 

•  Selection  of  solid  development  methods 

•  Application  of  rigorous  development  standards 

•  Insights  from  outside  of  development 

2.  Research  Needs 

•  Selection  of  solid  research  methods 

•  Application  of  rigorous  research  standards 

•  Insights  from  outside  of  research 

3.  Bridging  Research  and  Development  (examples  at  the  end> 

•  Decision-making  Tools — make  sense  of  research 
(models,  algorithms) 

•  Knowledge-making  Tools — make  sense  of  practice 
(learning  and  performance  assessment  tools, 
research  instruments)  13 


Inquiry  &  Measurement 

Strategy 

Understanding  Theoretical  Framework 

Team  Cognition 
Shared  Mental  Models 

Tools  Used  to  Understand  [Decision-making  &  Knowledge-making] 

Measuring 

Assessing 

Modeling 
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Team  Cognition — Framework 

Value 

1 .  Understand  Teams 

a.TC  has  potential  value  as  explanatory  mechanism 

2.  Predictive  Value 

a.  Construct  has  potential  to  be  a  predictive  variable  in 
teams 

3.  Diagnosis  Treatment 

a.  Help  to  diagnose  team  dysfunction  or  team 
excellence 

b.  Provide  insight  into  how  to  solve  issues 

4.  Facilitate  Interventions 

a.  Understanding  the  TC  behind  team  performance 
should  facilitate  intervention  design,  development, 


Team  Cognition — Implications 


Implications  of  Theoretical  Framework 

1 .  Team  Cognition  provides  a  framework  to  study 
performance  and  learning  teams 

2.  Team  Cognition  framework  provides  the  key  factors  that 
have  an  impact  on  team  outcomes 
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Team  Cognition — Concept 


•  More  than  the  sum  of  the 
individual  team  members 
cognition 


•  Emerges  from  the 
interaction  between 
individual  cognition  (each  team 
member)  and  team 

process  behaviors 
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Team  Cognition — Framework 


Collective  level 

Individual  knowledge 


Holistic  Level 

Taskwork/Teamwork  Knowledge 
— Long-term  &  Fleeting 


Team  Process  Behaviors 


Team  Knowledge 


I 

Team  Performance 


Team  Cognition — Types 


Types  of  Team  Cognition 

adaplad  from  Cooks  st  ai  300 0 


Team  Situational  Model 
v  Dynamic  trfTdareta'idJTg 


Team  Mental  Model 


Team  Knowledge 


Team.  Decision  Making 

Team  Cognition 

- P> 

Team  Situation  awareness 
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Mental  Models — Concept 

[aka:  Collective  cognition,  team  knowledge,  team  mental 
models,  shared  knowledge,  transactive  memory,  shared 

mental  models] 


Mental  Models 

“Mechanism  whereby  humans  generate  descriptions  of 
system  purpose  and  form,  explanations  of  system 
functioning  and  observed  system  state,  and  prediction  of 
future  system  State”  Rouse  &  Morris  (1986) 


24 


Shared  Mental  Model — 
What  do  we  mean  by  shared? 

1  .Shared  or  overlapping 

a.  Situations  where  two  or  more  team  members  having  ‘some  common 
knowledge’,  but  not  necessarily  ‘fully  redundant’ 

b.  Implying  a  knowledge  base  associated  with  the  work 

2. Similar  or  identical 

a. Team  members  holding  similar,  if  not  identical,  knowledge 

b. Most  directly  applying  to  shared  attitudes  and  beliefs. 

3.  Compatible  or  complementary 

a.  Not  need  shared  or  similar  knowledge,  but  knowledge  must  lead  team 
members  to  draw  similar  expectations  for  performance 

4. Distributed 

a.  Adequate  coverage  for  task  knowledge 

b.  Knowledge  being  effectively  apportioned  across  members 
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Shared  Mental  Model- 
shared”  Knowledge 


Shared  = 


and  Complementary 
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Shared  Mental  Model — Shared 

Knowledge  Types 


1. Task  Knowledge — domain  specific 

2. Team  Knowledge — 5  factors 

a. Team  Knowledge 

b. Team  Skills 

c. Team  Attitudes 

d. Team  Dynamics 

e. Team  Environment 
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Shared  Mental  Model — 
Link  to  Team  Performance 


Team  performance 
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Measuring  Team  Cognition — 

Why  Measure 

1  .Assessment  of  team  performance  or  effectiveness  (criterion 
or  dependent  measures)  often  taken  for  granted 

2.0utcome  measures  of  team  performance  do  not  typically 
reveal  why  performance  is  effective  or  ineffective 

3.  Process  measures  of  team  behavior  are  often  subjective 
and  lack  reliability  and  validity 

4. Team  cognition  is  assumed  to  contribute  to  team 
performance,  and  especially  for  growing  number  of  cognitive 
tasks 

5.  Understanding  the  team  cognition  behind  team  performance 
should  inform  interventions  (design,  training,  selection)  to 
improve  that  performance  34 
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Assessing  Team  SMM 

1  .Capture  conceptual  representations  of  individual  mental  models 

2.  Use  analytical  methods  to  compare  mental  models 

a.  Show  relationships  among  individuals  (team)  or  for  a  single  individuals  —  at 
one  point  or  over  time 

3.  Analytical  comparisons  are  carried  out  to  represent  differences  and 
change  of  cognitive  function 

4.  Research  Focus 

a.  Measure  Individual  Knowledge,  Determine  Shared  Knowledge 

b.  Measure  Team  Interactions,  Determine  Team  Patterns 

c.  Measure  Individual  &  Team  Performance,  Determine  Effects  of  Team 
Cognition 

5.  Research  Goal 

a.  What  are  the  patterns  and  underlying  mechanism  of  Team  Cognition 

6.  Workplace  Goal 

a.  Improve  Team  Performance 
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Techniques  to  Measure  Shared 

Knowledge 

1.  Traditional — measuring  concept  relatedness 

a. Card  sorting,  cognitive  interviewing,  MDS,  Pathfinder,  surveys, 

Casual  maps  (Langan-Fox,  Code,  Langfield-Smith,  2000;  Trochim,  1989) 

b.  Concept  Mapping — Statistical  analysis  &  Descriptive  analysis 

2.  Emerging  Methods 

a. Analysis  Constructed  Shared  Mental  Models  (AC-SMM) 

Methodology  (Johnson  &  O’Connor,  2004,  2008) 

b. Team-related  Knowledge  SMM  Instrument  (TADI)  (Johnson  et.  ai, 

2007) 

c. Surface,  Matching,  and  Deep  Structure  (SMD)  Methodology 

(Ifenthaler,  2006) 

d.  Model  Inspection  Trace  of  Concepts  and  Relations  (MITOCAR) 

Methodology  (Pirnay-Dummer,  2006) 
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Assessing  Team  SMM 

TAD  I 


Team  Assessment  and  Diagnostics  Instrument 
(TADI) 

Team-related  Knowledge  SMM  Instrument  (Johnson  et.  al,  2007) 
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TAD  I 


Team  Knowledge 


Team  Knowledge 
Factors 


Knowledge  about  team  members  and  tasks  that  they  need  to  perform 
Teammates  knowledge,  Task  knowledge 

Team  Skills 

Knowledge  about  team  members  abilities  associated  with  successful  job  performance 
Communication  skills,  Interpersonal  skills,  Leadership  skills,  Skills  to  deal  with  conflict 
and  team  cohesion 

Team  Attitudes 

Knowledge  about  team  members  internal  state  that  influences  team  members’  choices  or 
decision  to  act  in  a  certain  way  under  particular  circumstances 
Shared  belief,  Shared  value 

Team  Dynamics 

Knowledge  about  team  members  dynamic  processes  of  team  coordination  and  team 
cohesion 

Team  coordination,  Team  cohesion 

Team  Environment 


Knowledge  about  team  members  external  conditions  affecting  the  foundation  of  the  team 
mental  model 
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Technology,  Organization,  Synchrony  &  Geographic  dispersion 


Developing  Team  SMM 
Assessment  Techniques- 

HIMATT  Study 

Highly  Integrated  Model  Assessment 

Technology  &  Tools  (HIMATT)  (Johnson  et.  al,  2009) 
•Surface,  Matching,  and  Deep  Structure  (SMD) 
Methodology  (Ifenthaler,  2006) 

•Model  Inspection  Trace  of  Concepts  and  Relations 
(MITOCAR)  Methodology  (Pirnay-Dummer,  2006) 
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Similarity  Analysis 

HIMATT  based  on  graph  theory,  set  theory,  model  theory, 
and  similarity  distribution  measures  and  measurement  of 


Similarity  Dimension 

Similarity  Indices 

1  -Surface 

Structure 

2-Graphic 

3-Structure 

4-Gamma 

5-Concept 

Semantic 

6-Proposition 

7-Balance 

1  -sum  of  all  propositions;  2-diameter  of  spanning  tree  (quantity  of  links  for  the  shortest  path  for  most 
distant  notes — range  of  conceptual  knowledge);  3-not  graphical  but  mere  structure;  4-density  of 
vertices 
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HIMATT  Indices 

1  .Surface  measure  —  compares  the  number  of  vertices  within  two  graphs — values 
for  surface  complexity 

2.Graphical  matching  —  diameters  of  the  spanning  trees — indicator  for  range  of 

conceptual  knowledge  Measure  for  structural  complexity  only 

3.Structural  matching  measure  —  compares  complete  structures  of  two  graphs 

Measure  is  necessary  for  testing  all  hypotheses  that  make  assumptions  about  general  features  of  structure  (e.g., 
assumptions  which  state  that  expert  knowledge  is  structured  differently  from  novice  knowledge) 

4. Gamma  (density  of  vertices  measure)  —  quotient  of  terms  per  vertex  Since  both 

graphs,  (a)  those  which  connect  every  term  with  each  other  term  (everything  with  everything),  and  (b)  graphs  which  only 
connect  pairs  of  terms  can  be  considered  weak  models,  a  medium  density  is  expected  for  most  good  working  models 

5. Concept  matching  —  compares  the  sets  of  concepts  (vertices)  within  a  graph  to 
determine  the  use  of  terms  —  Counts  how  many  concepts  are  alike  Measure  is 

especially  important  for  different  groups  operating  in  the  same  domain  (e.g.  using  the  same  textbook).  It  determines 
differences  in  language  use  between  the  models 

6. Propositional  matching  —  value  compares  only  fully  identical  propositions 
(concept-link-concept)  between  two  graphs — semantic  similarity 

7. Balanced  semantic  matching  —  uses  concepts  and  propositions  to  match  the 
semantic  potential  between  the  knowledge  representations 
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Implications  and  Benefits — Team 

Assessment 

Future  Work 

♦  Team  Readiness  Assessment — realtime,  cyclical  nature  of 
job  tasks  (performance/recoil)  (performance  episodes) — 
how  ready  are  we? 

♦  Learning  feedback — how  am  I  doing?  how  are  we  doing? 

♦  Decision-making  tool — what  is  the  state  of  things?  where  do 
we  go? 

♦  Team  self  diagnosis — where  are  we  at?  what  to  focus  on? 

♦  Learning  &  Performance  Interventions — want  is  needed? 
what  types  of  tasks  are  critical  to  focus  on? 
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Agent-Based  Modeling 


Team  Process  Model  Vers- ion  100706 


Johnson  A  Rotcn berg  Kina 
teje-h  n  s  -  p  j&fs  u  .edu 


Implications  and  Benefits — Team  Modeling 

♦  Decision  making  tool 

♦  What  size  of  team  given  specific  team  and  task 
parameters 

♦  Diagnostic  tool 

♦  How  might  our  team  perform  given  our  team 
characteristics 

♦  Planning  Tool 

♦  How  much  will  out  team  get  out  of  a  specific  learning  task? 

♦  What  is  the  cost  of  having  too  many  teammates? 
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T earn  Assessment  and  Diagnostics  Instrument 
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